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1. Introduction 


The Australian Bureau of Statistics (ABS) is implementing a new approach to managing 
statistical processes called quality gates. Quality gates have been used in production 
industries for a long time but have only recently been applied to statistical processes. The 
ABS has adopted and expanded upon these ideas to create a risk mitigation tool for statistical 
processes. 


Quality gates in the ABS are designed to improve the early detection of errors or flaws in 
statistical processes (Schubert 2007). They act as a checkpoint at which an assessment of the 
quality is made, either qualitatively or quantitatively, to determine whether to proceed to the 
next stage in the process. Quality gates improve a user's ability to manage statistical risk by 
providing explicit evidence relating to the statistical process at a given point in time. They also 
facilitate improved knowledge management and information sharing. 


To distinguish quality gates from more general internal quality assurance checks the ABS 
specified six components for their construction. These six components are Placement, Quality 
Measures, Roles, Tolerance, Actions, and Evaluation. 


2. Placement 


"Placement" refers to the placement of quality gates throughout a statistical process. The 
placement of a quality gate should occur where a risk assessment of the process reveals that 
there is a need for a quality gate. Quality gates should be placed as early in the process as 
possible to allow early detection of any issues or errors. 

In determining where to place quality gates it is important to ask the questions: 

e Whatcan go wrong? 

e When can it occur? 

e What impact can it have? 


A basic map of the production processes that are to be monitored may be of use when 
determining the placement of quality gates. The map will help with assessing potential risks to 
the process by aiding understanding of how the system works and interacts with other 
systems and processes. 


Obvious places to consider for placement of a quality gate in statistical processes include: 
e Where hand-over or integration of data between multiple areas occurs; 

e When data transformations take place; and 

e Where then are changes to processes, methods and systems. 


3. Quality Measures 


Quality measures are a set of indicators that provide information about the quality of the 
statistical process at a specific point in time. When determining what quality measures to 
include in a quality gate it is important to consider the risks and what information is required to 
reveal if there is a problem with the process. Quality measures need to focus on both the 
immediate and end quality assurance requirements of the process. A quality gate may have 
multiple quality measures which assist in determining problems in the process. For example, 
quality measures could include the monitoring of key data items or series, response rates, 
non-response rates, imputation rates and validation checklists to name a few. 


{ 


4. Roles 


This component involves assigning tasks to various people or areas involved in the operation 
of a quality gate. The Roles component identifies areas or people who are directly connected 
to the quality gate and its operation, along with people or areas who are affected by issues 
with the process. It is important to make sure that people or areas dependent on the 
successful outcome of the process, who are not directly involved, are included in roles as 
stakeholders. This is so they can be informed of any issues identified from the quality gates 
that may impact on their work. 


The key roles for a quality gate have been identified as: 
e An operational person (gate keeper); 

e Stakeholders; and 

e Assign off person(s). 


The operational person, known as the gate keeper, is the person responsible for compiling the 
information for the quality gate. Their role includes documenting the quality gate and 
populating the information within the gate. The gate keeper is also responsible for ensuring 
that all roles are completed on time. 


Stakeholders can be either those directly involved in the process of supplying information for 
the quality gate, or those who have an interest in the outcome of the quality gate and 
subsequent data. 


A sign off person is responsible for signing off on the quality gate and endorsing the 
continuation of the statistical process to the next stage. It is recommended that the sign off 
person be independent of the compilation of the quality gate. Depending on organisational 
requirements there could be more than one sign off person for the quality gates in a statistical 
process. 


5. Tolerance 


Tolerance refers to an acceptable level of quality. The acceptable level could be qualitative 
(e.g. Yes/No) or quantitative (e.g. 97%) and is generally set by expectations of what should be 
observed at that point in the process for a given quality measure. Tolerance levels could take 
the form of thresholds. Thresholds provide a range of what is an acceptable level of quality. 


A powerful aspect of this component is that expectations of quality at a specific point in time 
are predetermined. This prevents an erosion of quality over time because potential minor 
problems or changes in quality will be identified. This may not be the case if expectations 
about the level of quality are not considered prior to the decision point. 


6. Actions 


Actions are predetermined responses to various outcomes for a quality gate. They provide a 
definition of what will be done if threshold or tolerance levels are met or not met with regards 
to each quality measure. In particular the actions associated with each quality measure need 
to take into account the severity of the result on the end product or other quality measures and 
gates if the threshold or tolerance levels are not met. 


Questions which may help clarify the actions to take for a quality measure depending on the 
tolerance levels are: 

e What needs to be done if there is a problem? 

e Who needs to be informed? 


The ABS uses a traffic light analogy to assist in determining the degree of severity of an action 
in response to the outcomes from a quality measure. This is where "green" represents that the 
expectations for the quality measure have been met. "Amber" occurs when the quality 
measure tolerance reveals that there may be an issue. Generally this requires further 
investigation into the potential problems whilst continuing cautiously onto the next stage of the 
process. Where the tolerance level has not been met a "red" light occurs. This means that the 
process must stop so that the issues can be investigated and resolved before proceeding to 
the next step. 


7. Evaluation 


Evaluation is the final component of a quality gate. At the end of each statistical process cycle 
is it recommended that the quality gates should be evaluated to determine what worked well, 
what didn't and where improvements can be made. It is useful to consider whether the 
information provided by the quality gates provided enough information to make informed 
decisions. Evaluation of quality gates consolidates the final reporting on the quality of the 
statistical process cycle. The subsequent documentation and knowledge management that 
occurs due to this explicit review of the entire statistical process cycle assists with maintaining 
confidence in the quality of the statistical output produced. 


8. The ABS experience in the implementation of quality gates 


Although the ABS has existing processes in place to quality assure its collections, as part of 
an ongoing commitment to continuous improvement the implementation of quality gates is 
gaining momentum across the agency. This has been assisted by areas in the ABS who were 
early adopters of quality gates who have provided feedback on their experiences with the 
implementation and use of the tool. The promotion and education of areas within the ABS on 
the use and implementation of quality gates has also assisted with their implementation. The 
experiences of both the areas implementing and the people responsible for promoting and 
championing quality gates have been very useful for their implementation. 


The championing of quality gates across the organisation has been a long process that has 
included the creation of training courses, workshops with individual areas, promotion through 
corporate meetings and general exploitation of any opportunities that arise with which to 
encourage and promote the use of quality gates. Often areas report that they don't have time 
to implement quality gates, or they already consider that they are 'doing enough’ checks to 
quality assure their processes. These challenges have been met with examples of where 
current processes have not been enough and how already existing documentation and checks 
can be utilised within quality gates to help in their creation. These examples and practical 
reuse of existing quality assurance checks have helped to promote the uptake of quality gates. 


People attempting to implement the tool have sometimes found the initial implementation of 
quality gates to be time consuming. This is partly due to areas needing to understand their 
processes but also because there is a learning curve for anyone using a new tool. The time 
taken for the initial creation of a quality gate can be quite long due to areas needing to 
understand the risks and expectations of their processes. In some cases this may be the first 
time that members of teams have considered their processes more broadly. However, the 
skills developed from the initial implementation of the first few quality gates provide invaluable 
knowledge which help in the development of subsequent quality gates which occur at a 
greater rate due to the previous experience. 


One of the common barriers that areas have found in the implementation of quality gates is 
creating too many quality gates. Areas have had difficulty in differentiating between placement 
and quality measures. Often areas will initially create a quality gate based on one quality 
measure because they have not established the placement of a quality gate which would 
house that quality measure and others in an appropriate place. In some cases areas have had 


duplicate quality gates created. That is, one for each of the stakeholder areas involved in the 
compilation of the quality gate. This has meant that there were too many quality gates created 
to be effective in the monitoring of the processes. 


The area responsible for the creation of Input, Output Tables noted that although the 
development and implementation of their first quality gate took a lot of time to finalise, 
approximately five weeks, the quality gate was extremely useful. The quality gate was being 
used to monitor processes whilst under development. The quality gate identified issues with 
the process which enabled the problem to be fixed earlier than would have otherwise 
occurred. The benefit of this error being picked up earlier in the process meant that the impact 
of the error wasn't as large as it would have been if left unidentified until later. 


The Input, Output section also noted that there were issues relating to the quality measures in 
terms of their number, assessment and viability. Their initial quality gate had too many quality 
measures. This meant that some quality measures were not able to be completely assessed 
within the necessary time frame before the process needed to move onto the next stage. 
Hindsight allowed the area to identify that they needed to make a call of an "amber" action 
more often and flag to continue to investigate in conjunction with the process continuing. 


It was also observed that some quality measures actually belonged in other quality gates in 
the process and not in the ones to which they were initially assigned. This identification 
became obvious when the area tried to assess these quality measures only to discover that 
there wasn't enough information at that stage in the process in order to complete the 
assessment. This was the case with quality measures that had exceptions to the rule which 
then required investigations to be undertaken to make sure that it was only the exceptions to 
the rule failing. For example the quality measures "exports do not exceed production" and "re- 
exports do not exceed imports" have exceptions in regards to agricultural commodities and 
reference period reporting respectively. This meant that the quality measures had to be moved 
to a later quality gate than where they were initially placed in order to have the required 
information available to help with the analysis. 


Other learning gained from the initial implementation included the identification of quality 
measures that needed to be expanded upon or removed from the quality gate. It was 
recognised that some quality measures were not viable because the time cost of having the 
quality measure outweighed its benefit. 


The area also learnt that the results from the quality measures should be documented and 
communicated with stakeholders of upstream processes to reduce the occurrence of issues 
being repeated due to upstream process deficiencies. 


The process of implementing the quality gates had positive side effects aside from identifying 
issues earlier and enabling them to be resolved in time for subsequent stages of the process. 
Knowledge management within the working groups improved. This was in part due to the 
documentation required for quality gates but also through individuals responsible for the 
compilation of various quality measures with the quality gates. These individuals obtained a 
greater understanding of the overall process and their part in it through the provision of 
information for the quality measures. 


9. Conclusion 


Quality gates are a useful tool for monitoring statistical processes. They provide a model of 
accountability along with expectations and actions to take depending on the situation. Quality 
gates also help with knowledge management of statistical processes. Although there are time 
costs associated with implementing quality gates initially, the benefits of the tool outweigh 
these costs. More detailed information on quality gates and their implementation can be found 


in the information paper "Quality Management of Statistical Processes Using Quality Gates, 
Dec 2010, (cat.no.1540.0)" on the ABS website <http://www.abs.gov.au/ausstats/abs@.nsf/mf/1540.0>. 
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